fix(codex): keep system prompts in input for GPT-5 automatic prompt caching - #1346

Problem

OpenAI's automatic prompt caching for GPT-5 models returns cached_tokens: 0 on every request routed through OmniRoute, even with identical 53K-token prompts sent seconds apart.

Root cause: The instructions field in the Responses API is not included in the prompt cache key computation for GPT-5 models. OpenAI only caches based on the serialized input array + tools. The current code moves system messages from input (cacheable) into instructions (not cacheable) via hoistSystemMessagesToInstructions(), destroying the cache prefix entirely.

Evidence

Diagnostic logging on a production deployment confirmed:

[cache-diag-resp] model=gpt-5.4 input=52889 output=370 cached=0 pct=0%   (request 1)
[cache-diag-resp] model=gpt-5.4 input=53421 output=29  cached=0 pct=0%   (request 2, 15s later)
[cache-diag-resp] model=gpt-5.4 input=53973 output=263 cached=0 pct=0%   (request 3, 50s later)

Three consecutive requests with identical instructions + tools prefix, same model, within 90 seconds — zero cache hits. Meanwhile, a MiniMax provider using the same proxy achieved cache hits immediately:

MiniMax request 1: cache_creation=10513, cache_read=0
MiniMax request 2: cache_creation=438,   cache_read=10513  ← cache hit!

This is consistent with community reports:

• Caching is borked for GPT 5 models
• No Caching with model Responses
• Problem caching system prompt

What this PR changes

For native Responses API passthrough requests (_nativeCodexPassthrough === true):

For translated requests (Chat Completions → Responses format): no change. The existing hoist + default instructions behavior is preserved.

New function: convertSystemToDeveloperRole()

Converts role: "system" → role: "developer" in-place within the input array. This is necessary because:

1. Codex rejects system role in input (existing comment in code confirms this)
2. GPT-5 models support developer role as the replacement
3. Keeping the content in input preserves the cacheable prefix

Impact

For a deployment routing ~7K daily Codex requests averaging 41K input tokens each:

• Before: 286.7M input tokens/day, 0.1% cache rate, ~8 cache hits total
• After (projected): Same volume, ~90% cache rate → ~258M cached tokens/day
• Cost reduction: ~90% on input token costs for cached portions
• Latency reduction: Up to 80% TTFT for cache hits (per OpenAI docs)

Testing

To verify the fix:

1. Deploy the updated executor
2. Monitor cached_tokens in response.completed SSE events
3. Second request with the same system prompt should show cached_tokens > 0

Example diagnostic (add to proxy/middleware):

// In response handler, look for response.completed events:
const u = completed.response?.usage;
const cached = u?.input_tokens_details?.cached_tokens ?? 0;
console.log(`cached=${cached} pct=${Math.round(cached / u.input_tokens * 100)}%`);